Systems to rate the strength of scientific evidence.

Health care decisions are increasingly being made on research-based evidence rather than on expert opinion or clinical experience alone. Systematic reviews represent a rigorous method of compiling scientific evidence to answer questions regarding health care issues of treatment, diagnosis, or preventive services. Traditional opinion-based narrative reviews and systematic reviews differ in several ways. Systematic reviews (and evidence-based technology assessments) attempt to minimize bias by the comprehensiveness and reproducibility of the search for and selection of articles for review. They also typically assess the methodologic quality of the included studies—i.e., how well the study was designed, conducted, and analyzed—and evaluate the overall strength of that body of evidence. Thus, systematic reviews and technology assessments increasingly form the basis for making individual and policy-level health care decisions. Throughout the 1990s and into the 21st century, the Agency for Healthcare Research and Quality (AHRQ) has been the foremost federal agency providing research support and policy guidance in health services research. In this role, it gives particular emphasis to quality of care, clinical practice guidelines, and evidence-based practice—for instance through its Evidence-based Practice Center (EPC) program. Through this program and a group of 12 EPCs in North America, AHRQ seeks to advance the field’s understanding of how best to ensure that reviews of the clinical or related literature are scientifically and clinically robust. The Healthcare Research and Quality Act of 1999, Part B, Title IX, Section 911(a) mandates that AHRQ, in collaboration with experts from the public and private sectors, identify methods or systems to assess health care research results, particularly “methods or systems to rate the strength of the scientific evidence underlying health care practice, recommendations in the research literature, and technology assessments.” AHRQ also is directed to make such methods or systems widely available. AHRQ commissioned the Research Triangle Institute–University of North Carolina EPC to undertake a study to produce the required report, drawing on earlier work from the RTI–UNC EPC in this area.1 The study also advances AHRQ’s mission to support research that will improve the outcomes and quality of health care through research and dissemination of research results to all interested parties in the public and private sectors both in the United States and elsewhere. The overarching goals of this project were to describe systems to rate the strength of scientific evidence, including evaluating the quality of individual articles that make up a body of evidence on a specific scientific question in health care, and to provide some guidance as to “best practices” in this field today. Critical to this discussion is the definition of quality. “Methodologic quality” has been defined as “the extent to which all aspects of a study’s design and conduct can be shown to protect against systematic bias, nonsystematic bias, and inferential error.”(Ref. 1, p. 472) For purposes of this study, the authors hold quality to be the extent to which a study’s design, conduct, Evidence Report/Technology Assessment Number 47